Stop gateway outages from paging java-chat as failed - #167
Merged
Conversation
A background keep-alive probe failure is a monitoring signal, not a user-facing request failure — the class doc already says so — yet the second consecutive failure logged ERROR. Every ERROR line feeds the critical java-chat-error-log Grafana alert, so a routine LLM gateway redeploy (minutes of 503s while the compose stack recreates) paged java-chat even though the app kept serving. Provider-health paging is owned by the dependencies health group (java-chat-dependencies-degraded) and the gateway's own availability rules, so the repeated-failure escalation now logs WARN like the first failure and the slow-probe loop. ERROR stays reserved for user-facing failures. Health still flips DOWN immediately; only the log level moves.
…ream dev.javachat.ai runs the dev profile, so logging.level.org.springframework.web=DEBUG shipped to Loki: a 5,000-line 3h sample held ~4,975 framework request-plumbing lines against 24 application lines (73,705 lines/24h vs production's 1,233), burying the EmbeddingModelKeepAlive events the java-chat Loki alert rules parse. Raise the level locally when debugging Spring MVC itself rather than for every deployed request.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughSummary by CodeRabbit
WalkthroughRepeated embedding probe failures now log at WARN instead of ERROR. The related test checks the new event level. The development profile removes default Spring MVC DEBUG logging and documents a temporary command-line override. ChangesEmbedding probe logging
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
coderabbitai
Bot
added
bug
There was a problem hiding this comment.
Pull request overview
This PR reduces noisy paging and log-volume in the dev deployment by (1) downgrading repeated background embedding keep-alive probe failures from ERROR to WARN (while still flipping actuator health to DOWN immediately), and (2) removing Spring MVC DEBUG logging from the dev profile so Loki/Grafana alerts and operational signals remain visible.
Changes:
- Change repeated embedding probe failure “loop” escalation from
ERROR→WARNinEmbeddingModelKeepAlive. - Strengthen tests to assert zero
ERRORlogs forembedding_model_probe_*events and to expect WARN for the loop event. - Remove
logging.level.org.springframework.web=DEBUGfromapplication-dev.propertiesand add rationale comments.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| src/main/java/com/williamcallahan/javachat/service/EmbeddingModelKeepAlive.java | Downgrades repeated probe failure escalation logging from ERROR to WARN and documents rationale. |
| src/test/java/com/williamcallahan/javachat/service/EmbeddingModelKeepAliveTest.java | Updates assertions to match WARN escalation and ensures no probe events emit ERROR. |
| src/main/resources/application-dev.properties | Stops shipping Spring MVC DEBUG logs in the dev profile and documents why. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+42
to
+51
| # More verbose logging for development. | ||
| # | ||
| # `dev` is also the profile the deployed dev.javachat.ai runs under, so anything set here ships | ||
| # to Loki. org.springframework.web at DEBUG was 99.5% of this service's log volume — a 5,000-line | ||
| # sample over 3h on 2026-08-04 held 1,991 DispatcherServlet + 1,990 HttpEntityMethodProcessor + | ||
| # 994 RequestResponseBodyMethodProcessor lines against 24 application lines, essentially all of it | ||
| # request plumbing for the 15s /actuator/prometheus scrape. It buried the signal that matters | ||
| # (EmbeddingModelKeepAlive, which the java-chat Loki alert rules parse) and cost 73,705 lines/24h | ||
| # against production's 1,233. Raise it locally when debugging Spring MVC itself, e.g. | ||
| # `--logging.level.org.springframework.web=DEBUG`, rather than for every deployed request. |
Comment on lines
+33
to
+37
| * <p>The escalation stays at WARN even when repeated: a background probe failure is a | ||
| * monitoring signal, never a user-facing request failure, and ERROR lines feed the | ||
| * critical java-chat-error-log Grafana alert. Provider-health paging is owned by the | ||
| * dependencies health group (java-chat-dependencies-degraded) and the gateway's own | ||
| * availability rules, so a routine gateway redeploy must not page here.</p> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fix(embeddings): keep the keep-alive failure escalation out of ERROR—EmbeddingModelKeepAlivelogged ERROR on the second consecutive background probe failure. Every ERROR line feeds the criticaljava-chat-error-logGrafana alert, so a routine LLM gateway redeploy (minutes of 503s while the compose stack recreates) paged java-chat while the app kept serving. The escalation now logs WARN, matching the first failure and the slow-probe loop; health still flips DOWN immediately. Provider-health paging is owned by the dependencies health group (java-chat-dependencies-degraded) and the gateway's own availability rules.fix(dev): drop DEBUG Spring MVC plumbing from the deployed dev log stream—logging.level.org.springframework.web=DEBUGshipped to Loki from dev.javachat.ai (~74k lines/24h, 99.5% framework plumbing), burying theEmbeddingModelKeepAliveevents the java-chat Loki alert rules parse.Pairs with WilliamAGH/grafana-config
2afc20a, which revived the silently-deadjava-chat-error-logrule (its| jsonpipeline dropped every raw log line after the Alloy envelope unwrap).Test plan
./gradlew testforEmbeddingModelKeepAliveTest+EmbeddingModelKeepAliveSchedulerTest(8 tests, green; updated assertion expects WARN, zero ERROR)JavaChatApplicationTestsgreen599118c4), Coolify dev deploy finished and verified healthy